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METHOD AND APPARATUS FOR PROVIDING PEER-TO-PEER FORCE 
FEEDBACK OVER A COMPUTER NETWORK 



BY INVENTORS 

5 Louis B. Rosenberg, Scott B. Brave and Sian W. Tan 



CROSS-REFERENCE TO RELATED APPLICATIONS 

The present application is a continuation-in-part of U.S. Patent Applications: 
Serial No. 09/050,665, entitled "Method and Apparatus for Controlling Force 
Feedback Interface Systems Using a Host Computer", filed March 30, 1998, which is a 
continuation of Patent No. 5,734,373, filed December 1, 1995; 

Serial No. 08/571,606, entitled "Method and Apparatus for Providing Force 
Feedback for a Graphical User Interface," filed December 13, 1995; 

Serial No. 08/691,852, entitled "Method and Apparatus for Providing Force 
Feedback over a Computer Network," filed August 1, 1996; and 

Serial No. 08/664,086, entitled "Method and Apparatus for Providing Simulated 
Physical Interactions within Computer Generated Environments," filed June 14, 1996, 
which claims priority of provisional application serial no. 60/017,803, filed May 17, 
1996, 

all of which are incorporated herein by reference for all purposes. 



Background of the Invention 

This invention relates generally to human/computer interfaces, and more 
particularly to human/computer interfaces with force feedback that can operate over a 
25 network. 

Computer networks have become essential to allow users and computers to 
communicate with each other. Users transmit and receive data over networks in offices, 
at home, or in portable devices, and do so for a variety of tasks and applications, 
including communication, distribution, and entertainment. Many different types of 
30 networks are used. Local Area Networks (LANs) are typically provided in a limited area 
and include a relatively small number of computer nodes. The most large scale example 
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of a network is the Internet, which has become extremely popular. The Internet is a Wide 
Area Network (WAN) that is structured and distributed such that no one authority or 
entity manages the network. Different communication protocols can be used over 
networks to allow computers to communicate with each other; for example, protocols 
5 such as "Transmission Control Protocol/Internet Protocol" (TCP/IP) and the "World 
Wide Web" (WWW) are used over the Internet. TCP/IP sends "packets" of data 
between a host machine, e.g. a server computer on the Internet, and a client machine, e.g. 
a user's personal computer connected to the Internet, or between two client machines. 
The WWW is an Internet interface protocol which is supported by the same TCP/IP 
10 transmission protocol. Intranets are private networks based upon Internet standards, and 
have become quite common for managing information and communications within an 
organization. Intranets, since they adhere to Internet standards, can often use the same 
interface software as are used on the Internet, such as a web browser. 

A variety of information is currently transferred over computer networks. For 
15 example, visual data, text data, and sound data can be transmitted over the Internet and 
the WWW. Image data can be sent from one client machine to another (or from a server 
machine to a client machine) in a variety of formats. Or, for example, data packets coded 
in TCP/IP format can be sent from one client machine to another over the Internet to 
transmit sound data. This last-mentioned technique forms the basis for Internet 
20 telephony. 

While the transmission of visual images (both static and dynamic, i.e. video), text, 
and sound over networks, such as the Internet, is well-known, the transmission of other 
types of sensory data has not been well explored. In particular, the transmission of data 
over networks pertaining to the sense of touch and/or force has not been established. 
25 "Force feedback" allows a user to experience or "feel" tactile sensations as provided 
through computational information. Using computer-controlled actuators and sensors on 
a force feedback device, a variety of realistic sensations can be modeled and experienced 
by the user. This useful and highly immersive sensory modality for interacting with the 
Internet and other users over the Internet has hereto been unavailable. 
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Summary of the Invention 



The present invention is related to the transmission of information pertaining to a 
subset of the sense of touch, i.e. the transmission of forces to a user over a computer 
5 network system. The " force feedback" provided by the methods and apparatus of the 
present invention enhance the sensory experience of user-to-user interactions to provide a 
richer, more interesting, and more enjoyable experience. 

A network force feedback system in accordance with the present invention 
includes a network, a first computer coupled to the network, and a second computer 

10 coupled to the network. The first and second computers each include a visual display and 
a force feedback interface device. The interface device is capable of providing a 
computer input to the computer and also includes an actuator to output force feedback to 
a user in response to a force feedback signal provided by the computer. At least one of 
the computers develops an image on the visual display that is associated with stored force 

15 feedback information, and produces the image and the force feedback signal based on 
information received from the other, remote computer. Preferably, the computers 
produce the images and the force feedback signals based on information received from 
the remote computer and based on the computer input from the local force feedback 
device. The force feedback device can include a local microprocessor that communicates 

20 with the computer such that the force feedback signal can take the form of a relatively 
high-level force command. The present invention therefore permits two computer users 
to interact using force feedback provided over a network on a client-to-client (peer-to- 
peer) basis. 

A method for providing force feedback between two computers over a network 
25 includes establishing a connection between a first computer and a second computer over a 
network, sending first computer information to a second computer from the first 
computer over the network, and providing a force feedback signal to the second force 
feedback device from the second computer, where the force feedback signal is based on 
the first computer information. The force feedback signal causes the second force 
30 feedback device to output forces to the second user using an actuator of the force 
feedback device. Similarly, second computer information is sent to the first computer 
from the second computer over the network, and a force feedback signal is provided to 
the first force feedback device from the first computer. The force feedback signal is 
based on the second computer information, where the force feedback signal causes the 
35 first force feedback device to output forces to the first user using an actuator of the first 
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force feedback device. The force feedback signal can also be based on the input 
information from the force feedback devices. The information sent over the network can 
include position information describing a position of a manipulandum of the force 
feedback devices, and/or can include force feedback information indicating a force 
5 sensation to be output by the remote force feedback device. The computers can each 
display a graphical environment having a first graphical object controlled by the first user 
and a second graphical object controlled by the second user. 

In a different aspect, a method is disclosed of allowing two users to interact 
physically over a computer network, wherein a first manipulandum is physically 

10 contacted and moved by a first user in at least one degree of freedom and a second 
manipulandum is physically contacted and moved by a second user in at least one degree 
of freedom. First information is transmitted over the computer network, including an 
indication of movement of the first manipulandum, to a second manipulandum physically 
contacted by a second user. A force is applied to the second manipulandum based on the 

15 indication of movement of the first manipulandum such that the second user feels an 
interaction based on movement of the first manipulandum. Second information is 
similarly transmitted to the first manipulandum such that the first user feels an interaction 
based on movement of the second manipulandum. Two users can thus physically 
exchange information and interact over a computer network. 

20 The present invention adds a new sensory modality when interacting with a 

networked computer system. More particularly, force information can be either 
downloaded to a client machine from a server machine connected to the network, or force 
information can be passed between two or more client machines on the network. Peer-to- 
peer or server-to-peer direct interaction allows two or more users to interact using a 

25 different sensory modality, the sense of touch. The interaction may be subject to some 
transmission ("latency") delays on networks such as the Internet, but permits remote 
interactivity with a client's force feedback device in new ways to enhance communication 
and interaction between users. 

These and other advantages of the present invention will become apparent upon 
30 reading the following detailed descriptions and studying the various figures of the 
drawings. 
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Brief Description of the Drawings 



Fig. 1 is a pictorial representation of the Internet, a web server machine, and two 
client machines; 

5 Fig. 2 is a block-diagram of a client machine used in the present invention; 

Fig. 3 is a block-diagram of a force feedback system in accordance with the 
present invention; 

Fig. 4a is a perspective view of a preferred human/computer interface ("force 
feedback device") of the present invention; 

10 Fig. 4b is a cross-sectional view taken along line 4b-4b of Fig. 4a; 

Fig. 5a is a perspective view of another preferred embodiment for a force 
feedback device in accordance with the present invention; 

Fig. 5b is a perspective view of a first alternate embodiment for the force feedback 
device of Fig. 5a; 

15 Fig. 5c is a perspective view of a second alternate embodiment of the force 

feedback device of Fig. 5a; 

Fig. 6 is a block diagram of a wide area network (WAN) based upon Internet 
TCP/IP protocol and supporting World Wide Web (WWW) HTML protocols in 
accordance with the present invention; 

20 Fig. 7a is flow-diagram of a "Acquire URL" process in accordance with the 

present invention; 

Fig. 7b is an example of an HTML file of the present invention sent from a web 
server machine to a client machine; 

Fig. 8 is a flow-diagram of the "Parse and Interpret HTML Component" step of 

25 Fig. 7a; 

Fig. 9 is a flow-diagram of the "Plug-in Interpret .IFF File" step of Fig. 8; 

Fig. 9a is an illustration of an image displayed on a visual display of a client 
computer as generated from a downloaded HTML web page file; 
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Fig. 10 is a flow-diagram of a process for monitoring the "pointer state" of the 
force feedback device of the present invention; 

Fig. 11 is a schematic diagram of a multi-computer network system used in 
accordance with client-to-client embodiments; and 

5 Figs. 12a and 12b are diagrammatic illustrations of a client-to-client game 

embodiment displayed on a display screen of a client computer. 
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Detailed Description of the Preferred Embodiments 



In Fig. 1, a network system 10 includes a network and a number of computers or 
"machines" coupled to the network. In the described example of Fig. 1, the network is 
5 the Internet 12. For example, a first client machine 14, a second client machine 16, and a 
web server machine 18, are coupled to the Internet 12. Although embodiments specific to 
the current form of the Internet are described, it should be appreciated that the present 
invention can also be used in conjunction with many different types of networks (any 
LAN or WAN) using appropriate communication protocols. Such types of networks and 
1 0 the communications used over such networks are well known to those skilled in the art. 

As noted previously, both the Internet 12 and Intranets operate using the same 
TCP/IP protocols. This allows Intranets to use similar or the same server machine 
software and client machine software as are used in Internet 12 applications. Therefore, it 
will be apparent to those skilled in the art that the following descriptions apply equally 
15 well to Internet, Intranet, and other forms of network systems that are compatible with the 
processes and apparatus disclosed herein. 

The Internet 12 includes a number of nodes 20 that are interconnected by data 
transmission media 22. These nodes are typically routers, switches, and other intelligent 
data transmission apparatus which route "packets" of TCP/IP information to the desired 
20 destination. In some instances, the nodes 20 comprise an Internet service provider (ISP) 
20a which allows a client machine to access the "backbone" of the Internet. 
Alternatively, client machines and web servers can be coupled directly into the backbone 
of the Internet. 

As noted previously, the present invention is directed to the implementation of 
25 force feedback over a network, such as the Internet 12. To provide a user of a client 
machine with the experience of force feedback, force feedback human/computer 
interfaces (hereafter "force feedback devices") 24 and 26 can be provided as part of the 
client machines 14 and 16, respectively. The client machines 14 and 16 are typically 
provided with computer video monitors 28 and 30 (which is one example of a "visual 
30 display"), respectively, which can display images II and 12, respectively. Preferably, 
forces developed by force feedback devices 24 and 26 are correlated with the images II 
and 12 of the client machines 14 and 16, respectively. 

The machines 14-18 are considered, in the language of the Internet, to be 
"resources," and each has its own unique Uniform Resource Locator or "URL." In one 
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embodiment of the present invention, a client machine, such as client machine 14 or 16, 
sends a request for a "web page" residing on, for example, web server machine 18. This 
is accomplished by the client machine sending a connection request and a URL which 
specifies the address of the web page to the web server machine 18. The web server 
5 machine 18 then sends a web page 32 written in HTML format back to the requesting 
client machine where it is " cached" in the memory (typically the RAM, hard disk, or a 
combination of the two) of the client machine. In this embodiment of the invention, the 
image on the video display of the client machine is generated from the HTML web page 
file cached on the client machine, and force feedback is provided to a user through the 
10 force feedback device as he manipulates a user manipulable object of the force feedback 
device. 

In a peer-to-peer aspect of the present invention, a first client machine, such as 
client machine 14, and a second client machine, such as client machine 16, directly 
communicate force feedback commands to each other in standard TCP/IP protocol over 

15 the Internet 12. More particularly, client machine 14 can send force feedback and other 
information to the URL of the client machine 16, and the client machine 16 can send 
force feedback and other information in standard TCP/IP packets to the URL of the client 
machine 14. In this way, users of client machine 14 and client machine 16 can interact 
physically over the Internet 12. Of course, a server machine 18 can likewise directly 

20 communicate force feedback commands to a client machine 12 or 14, or all three 
machines can interact. The client machines can also communicate directly over other 
types of networks and/or using other communication protocols. Peer-to-peer (i.e. client- 
to-client) communication is described below with reference to Fig. 11. 

In Fig. 2, a "personal" computer 34 architecture that can be used for client 
25 machine 14 or client machine 16 is shown in block diagram form. It should be noted that 
a variety of machine architectures can be used to access the Internet 12, i.e. can be used as 
"network access computers." The particular architecture shown for the computer 34 is a 
typical personal or "PC" computer architecture, such as that used with IBM compatible 
personal computers. Web server machines can also have similar architectures, but are 
30 often more powerful computers known as "workstations" that operate under some 
variant of the UNIX® operating system. The Internet service providers 20a are likewise 
often UNIX-based computers or powerful personal computers running Windows NT®. 
The nodes 20 are most commonly routers built by Cisco Systems of San Jose, California. 
Client machine 14 or 16 can also take other forms, such as a television including or 
35 connected to a microprocessor for Internet access, or a video game console system such 
as those available from Nintendo, Sega, or Sony. Force feedback devices used with such 
client machines can be appropriate for the particular embodiment, e.g., a TV remote 
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control used for internet browsing on the abovementioned television can include force 
feedback functionality. 

The personal computer system 34 includes a microprocessor 36 clocked by a 
system clock CLK and which is coupled to a high speed or memory bus 38 and to a lower 
5 speed or I/O bus 40. The system RAM 42 and ROM 44 are typically coupled to the high 
speed memory bus, while various peripherals, such as the video display, hard disk drive, 
Internet interface (often either a modem or an Ethernet connection), and force feedback 
device, are typically coupled to the slower I/O bus. The microprocessor executes 
programs stored in the various memories (RAM, ROM, hard disk, etc.) of the personal 
10 computer 34 to control, for example, the image display on the video display and the 
forces provided by the force feedback device. The manufacture and use of personal 
computers, such as personal computer 34, are well-known to those skilled in the art. 

In Fig. 3, a client machine 46 in accordance with the present invention includes a 
personal computer system 48 and a force feedback human/computer interface or " force 

15 feedback device" 50. A user 52 can receive visual information 54 and auditory 
information 56 from the personal computer 48 and can manipulate the force feedback 
device 50 as indicated at 58a and 58b to provide input, e.g., to command a cursor location 
on a visual display or other provide other control information. In addition, the user 52 
can receive force feedback 60 from the force feedback device 50 to represent physical 

20 "feel" sensations. 

The personal computer system 48 includes the microprocessor 36, the system 
clock 62, a video monitor 64 (which is one type of "visual display"), and an audio 
device 66. The system clock 62, as explained previously, provides a system clock signal 
CLK to the microprocessor 36 and to other components of the personal computer system 
25 48. The display device 64 and the audio output device 66 are typically coupled to the I/O 
bus 40 (not shown in this figure). 

In this preferred embodiment, the force feedback device 50 preferably includes a 
local microprocessor 68, a local clock 70, optional local memory 71 for the local 
microprocessor 68, a sensor interface 72, sensors 74, a user manipulatable object 76, 
30 "other" input interface 78, an actuator interface 80, a safety switch 82, and actuators 84 
which provide a force F to the object 76, and an optional power supply 86 to provide 
power for the actuator interface 80 and actuator 84. 

The microprocessor 36 of the personal computer system 48 is coupled for 
communication with the local microprocessor 68 of the force feedback device 50. This 
35 communication coupling can be through a serial port coupling 88 to the personal 
computer system, or through a game port coupling 90 to the personal computer system. 
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Virtually all personal computer systems built to the IBM PC/AT standards will include a 
serial port and a game port. As noted, the serial port will permit two-way communication 
between microprocessor 36 and microprocessor 38, and thus is preferable over the game 
port coupling which only permits one-way communication from the local processor 68 to 
5 the microprocessor 36. In consequence, a serial port connection between the personal 
computer system 48 and the force feedback device 50 will permit force feedback 
commands to be sent from the microprocessor 36 to the local microprocessor 68, while a 
game port connection alone will not be able to provide this function. However, some 
simpler forms of "reflex" type force feedback can still be provided by the force feedback 

10 device 50 under the control of the local microprocessor 68 even if only a slower interface 
is used. It should also be noted that the microprocessor 36 and a local microprocessor 68 
may communicate over both the serial port and game port connection to provide a greater 
communication bandwidth. A preferred serial port is the Universal Serial Bus (USB) of a 
personal computer, although an RS-232 serial bus, or other serial busses, a parallel bus, 

15 an Ethernet bus, or other types of communication links can also be used. 

In use, the user 52 of the client machine 46 grasps the user object 76 (or 
"manipulandum") of the force feedback device 50 and manipulates {i.e. exerts a force to 
move or attempt to move) the user object to cause a "pointer" or other graphical object to 
move in the image displayed by the display device 64. For example, a pointer typically 

20 takes the form of a small arrow, a pointing hand, or the like. The sensor 75 senses the 
movement of the user object 76 and communicates the movement to the local 
microprocessor 68 through the sensor interface 72. The local microprocessor 68 then 
communicates through serial port 88, game port 90, or both to the microprocessor 36 to 
cause the microprocessor 36 to create a corresponding movement of the pointer on the 

25 image displayed upon the visual display 64. In some embodiments, the sensors 74 can 
communicate directly to microprocessor 36 without the use of local microprocessor 68. 
The user can also create other input, such as a "button click," through the other input 78 
which are communicated to the microprocessor 36 by the local microprocessor 68 or 
directly, e.g., using a game port. The user object 76 can take many forms, including a 

30 joystick, mouse, trackball, steering wheel, medical instrument, representation of a body 
part, gamepad controller, etc., as described in Patent Nos. 5,734,373 and Application 
Serial Nos. 08/664,086 and 08/881,691, all incorporated by reference herein. 

If the pointer on the display device 64 is at a position (or time) that correlates to a 
desired force feedback to the user 52, or an event occurs that dictates that force feedback 
35 should be output, the microprocessor 36 sends a force feedback command to the local 
microprocessor 68 over the serial port connection 88. The local microprocessor 68 parses 
this force feedback command and sends signals to the actuator interface 80 which causes 
the actuator 84 to create forces F on user object 76, which are experienced by the user 52 
10 
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as indicated at 60. The safety switch 82, sometimes referred to as a "deadman switch", 
blocks the signal from the actuator interface 80 if, for example, the user 52 is no longer 
grasping the object 76. In this way, the user 52 can interact with the client machine 46 in 
a visual, auditory, and tactile fashion. 

5 For example, when using the local microprocessor 68 to offload computational 

burden from the host computer, the host can send high level commands to the local 
microprocessor 68. The local microprocessor can parse or interpret the commands and 
implement a local force routine that is stored in local memory 71. Such a force routine 
might instruct the microprocessor 68 to read sensor positions, determine a force based on 

10 the sensor positions, and command the actuators 84 to output the force, all in a local 
control loop independent from the host computer (the microprocessor 68 would also 
preferably relay the sensor positions to the host computer). Different force routines can 
be provided to command different types of force sensations (spring forces, damping 
forces, vibration forces, etc.). This local control loop can be helpful in increasing the 

15 response time for forces applied to the user object 76, which is essential in creating 
realistic and accurate force feedback. The hardware architecture described above is also 
described in U.S. patent 5,739,811, and the high level command protocol between the 
computer and the force feedback device is also described in U.S. patent 5,734,373, the 
disclosures of which are incorporated herein by reference. 

20 In addition to sending force feedback commands, it may be convenient for host 

computer 48 to send a "spatial representation" to microprocessor 68, which is data 
describing the layout of all or some of the graphical objects displayed in the hosts' 
application program or graphical environment which are associated with forces and the 
types of these graphical objects (in the Web page embodiment, the layout/type of 

25 graphical objects can be downloaded from a remote computer providing the Web page). 
The microprocessor 68 can store such a spatial representation in memory 71, for example. 
In addition, the microprocessor 68 can be provided with the necessary instructions or data 
to correlate sensor readings with the position of the cursor on the display screen. The 
microprocessor would then be able to check sensor readings, determine cursor and target 

30 positions, and determine output forces independently of host computer 48. The host can 
implement operating system functions (such as displaying images) when appropriate, and 
low-speed handshaking signals can be communicated between processor 68 and host 48 
to correlate the microprocessor and host processes. Also, memory 71 can be a permanent 
form of memory such as ROM or EPROM which stores predetermined force sensations 

35 (force models, values, reflexes, etc.) for microprocessor 68 that are to be associated with 
particular types of graphical objects. 
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The host can also send the microprocessor a positional offset that may have 
occurred between the graphical object or user object controlled by the user and the 
graphical object or user object controlled by a remote user in a game or simulation. The 
microprocessor can use the positional offset in the determination of forces. For example, 
5 a spring force can be implemented between the user manipulatable objects of two 
networked host computers, where the magnitude of the spring force is proportional to the 
positional offset between the two user objects. The spring force thus biases the user 
objects to synchronized positions. 

In Fig. 4a, a force feedback device 50a is provided with a user manipulatable 
10 object 76a which, in this instance, includes a shaft 90 and a ball-grip (or joystick) 92. 
The force feedback device 50a also includes a pair of linear voice coil actuators ("voice 
coils") 94 and 96 that can serve both as sensors and actuators. Alternatively, the voice 
coils can be used only as actuators, and separate sensors (not shown) can be used. The 
voice coil 94 is coupled to the shaft 90 of object 76a by a first link 98 and a second link 
15 100. Link 98 is coupled to link 100 with a pivot 102, and a link 100 is coupled to the 
shaft 90 by a pivot 104. Similarly, voice coil 96 is coupled to the shaft 90 of the object 
76a by a first link 106 and a second link 108. The first link 106 is coupled to second link 
108 by a pivot 110, and the link 108 is coupled to the shaft 90 of the object 76a by the 
pivot 104. 

20 The link 98 can move in and out of a housing 1 12 as indicated by arrow 114, and 

link 106 can move in and out of a housing 116 of voice coil 96 as indicated by the arrow 
118. The pivots 102, 104, and 110 allow the object 76a to move within the constraints of 
an x-y plane, but does not permit movement in a z direction orthogonal to the x-y plane. 
Therefore, the force feedback device is a two degree (2D) of freedom device. That is, the 

25 user manipulatable object 76a can move with a first degree of freedom in a x direction, 
and in a second degree of freedom in the y direction. A 2D force feedback device 50a is 
considered preferable in the present invention since it correlates well to the two- 
dimensional screen of a monitor of a client machine. 

In Fig. 4b, a voice coil 94 is shown in a cross sectional view taken along line 4b- 
30 4b of Fig. 4a. The housing 112 includes a central core 120 and a number of elongated 
magnets 122. An armature 124 includes a hollow, cylindrical member having inner 
surface 126 which slidingly engages the core 120. Wrapped around the armature 124 are 
coils 128. The coils are electrically coupled to actuator and/or sensor interfaces. A plate 
1 30 is attached to the end of the armature 124 and is coupled to the link 98. The armature 
35 124 and link 98 can move in a linear fashion as indicated at 114. Other voice coil 
configurations can also be used, such as differently shaped cores, different coil layouts, 
etc. 
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The force feedback devices of Figs. 4a and 4b are also described in U.S. patent 
5,805,140, the disclosure of which is incorporated herein by reference. In particular, the 
operation of the voice coils as actuators and/or sensors is described therein. 

In Fig. 5 a, an alternative embodiment of a force feedback device 50b is illustrated. 
5 The force feedback device 50b has many points of similarity with the force feedback 
device 50a, with like reference numerals referring to like elements. The force feedback 
device 50b includes the user manipulatable object 76a, the shaft 90, the ball knob 92, and 
the voice coils 94 and 96. However, the links of the force feedback device 50a have been 
replaced by flexure members. More particularly, the links 98 and 100 of force feedback 

10 device 50a have been replaced by a rigid connector 132 and a flexible member 134 
(collectively comprising a "flexure member"), and the links 106 and 108 of the force 
feedback device 50a have been replaced by a connector member 136 and a flexible 
member 138 (also collectively comprising a flexure member). The connector 132 is 
rigidly is attached to the plate 130 at the end of the armature of the voice coil 94 and is 

15 rigidly attached to an end of the flexible member 134. The other end of the flexible 
member 134 is attached to a base 140 which, in turn, is rigidly attached to the shaft 90 of 
the object 76a. Similarly, the connector 136 is attached to a plate of an armature of voice 
coil 96 at one of its ends, and is attached to the flexible member 138 at the other of its 
ends. The remaining end of flexible member 138 is rigidly attached to the base 140. 

20 The flexible members 134 and 138 serve the same functions as the links of the 

force feedback device 50a described previously. As the object 76a is moved back and 
forth along an x-y plane, the flexible member 134 can move in and out of the voice coil 
housings 94 and 96, respectively, and can bend to accommodate angular movement with 
respect to the x and y axis. This permits the connectors 132 and 136 to move back and 

25 forth within the voice coils 94 and 96, respectively. The force feedback device of Fig. 5 a 
is also described in U.S. patent 5,805,140, the disclosure of which has been incorporated 
herein by reference. 

In Fig. 5b, an alternative user manipulatable object 76a takes the form of a stylus 
142 which can engage an aperture 144 in an alternative base 140'. The alternative base 
30 140' can be coupled to the flexible members 134 and 138 of the embodiment of Fig. 5a. 
Alternatively, the tip of stylus 142 can be rigidly or rotatably attached to alternative base 
140' with, for example, a ball joint or other joint or fastener. 

In Fig. 5c, another alternative base 140" is provided with an enlarged aperture 
144' which can be engaged by the tip of a finger 146 of the user. The base 140" then 
35 becomes the user manipulatable object 76c. As before, the base 140" is coupled the 
flexible members 134 and 138 of the first feedback device 50b of Fig. 5a. The structures 
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of Figs. 5b and 5c are also described in U.S. patent 5,721,566, the disclosure of which is 
incorporated herein by reference. 

The embodiments of figures 5b and 5c illustrate two of a range of equivalent user 
manipulatable objects suitable for the present invention. It should be apparent to those 
5 skilled in the art that these alternative objects 76b of Fig. 5b and 76c of Fig. 5c can 
equally well be used with other force feedback devices, such as the force feedback device 
50a illustrated in Fig. 4a. 

As noted previously, a preferred embodiment of the present invention provides a 
user manipulatable object that has two degrees of freedom. Other user manipulatable 
10 objects having one degree of freedom or three or more degrees of freedom are also within 
the scope of the present invention. For example, one embodiment of the present 
invention provides only one degree of freedom. Other force feedback devices of the 
present invention include mice, joysticks, joypads, a steering wheel, and yolks having 
two or more degrees of freedom. 

15 In Fig. 6, a conceptual representation of the network system 10 with force 

feedback includes a server machine 18, a client machine 14 provided with a force 
feedback device 24, and one or more additional client machines 16, each of which may be 
provided with additional force feedback devices 26. As noted in this figure, the server 
machine is a computer or "processor" running, for example, the TCP/IP server software 

20 and is which is connected to the Internet. The client machine 14 includes a computer or 
"processor" running Internet browser software and force feedback driver software. The 
processor of the client machine is connected to the Internet and to the force feedback 
device 24. The force feedback device 24 has sensors and actuators so that it can track 
movement of the user manipulatable object, monitor for button presses and/or other 

25 ancillary input devices, and provide output force feedback sensations. The force 
feedback device 24 sends object tracking information to the client machine, and receives 
force feedback commands from the client machine 14. The "additional client", such as 
client machine 16, also includes computers or "processors" running Internet browser 
software and force feedback driver software. The processors of these additional clients 

30 are also connected to the Internet and are connected to force feedback devices associated 
with that client. 

As noted in Fig. 6, a client machine 14 can send a data request to the server 
machine 18 and, in return, receive an HTML web page file including a special file of the 
present invention known as an "IFF" file. As will be appreciated by those skilled in the 
35 art, the server must also have a modified configuration file which lets it know that .IFF is 
a valid MIME type. This modified file would be a SRM.CONF or other .CONF file. The 
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client machine 14 then sends force feedback commands to the force feedback device 24 
and receives tracking and button data from the force feedback device 24. Client machine 
16 can likewise send a data request to the server machine 18 and receive an HTML file 
with one or more IFF files. The client machine 16 can then interact with the force 
5 feedback device 26 by sending force feedback commands to the device 26 and by 
receiving tracking and button data from the force feedback device 26. 

In addition to communicating with the server machine, the client machines can 
communicate directly with each other over the Internet using an Internet communication 
protocol. For example, client machine 14 can communicate with client machine 16 

10 through a TCP/IP connection. This is accomplished making the URL of the client 
machine 16 known to the client machine 14, and vice versa. In this fashion, direct 
communication between client machines can be accomplished without involving the 
server machine 18. These connections can send force feedback information and other 
information to the other client machine. For example, a process on the client machine 16 

15 can send force feedback information over a TCP/IP Internet connection to the client 
machine 14, which will then generate a force feedback command to the force feedback 
device 24. When the user reacts to the force feedback at force feedback device 24, this 
information can be sent from client machine 14 to client machine 16 to provide force 
feedback to the user on force feedback device 26. 

20 In Fig. 7a, a flow diagram illustrates an "acquire URL" process 146 running on a 

client machine, such as client machine 14 or client machine 16. This process is used 
when a client downloads a web page and force information from a server machine. This 
process 146 can be implemented using a standard Internet browser with a "plug-in" 
extension which permit the handling of force feedback commands. A preferred browser 

25 software is Netscape Navigator® software available from Netscape Corporation of 
Mountain View, California. The plug-in software is a proprietary extension of the web 
browser software, where this proprietary extension was developed by the Applicant of the 
present application. 

The process 146 begins at 148 and, in a step 150, a connection request is sent to 
30 the "host" of the desired URL. The host, in this example, is a server machine 18 and the 
desired URL is the URL of the desired web page residing on the server machine 18, the 
web page including force feedback commands. Alternatively, the desired web page can 
reside on another server or resource and be retrieved by server machine 18. In response 
to the connection request of step 150, the server machine 18 sends the HTML file 
35 representing the web page over the Internet to be received by the client machine. The 
HTML file includes a number of "components" which are typically commands, 
command fragments, instructions, and data which permit the display of the web page and 
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other web browser functionality. In a step 154, and an HTML component is obtained. If 
this component is the end of file ("eof ), a step 156 detects that fact and the process is 
completed at 158. Otherwise, the HTML component is parsed and interpreted at a step 
160 and process control is returned at step 154. It should be noted that most web browser 
5 software will start parsing and interpreting {i.e. processing) the HTML components even 
before the entire HTML file is received at the client machine. Alternatively, the entire 
HTML file can be received before the processing begins. 

In Fig. 7b, an example of an HTML web page 32, sent from a web server machine 
18 to a client machine (such as client machine 14 or 16) over the Internet 12, is shown. 

10 The HTML file 32 includes a number of "components" which are parsed and interpreted 
as previously described. An HTML file begins with a <HTML>command 162 to indicate 
the start of the HTML file, and a <BODY> command 164 to indicate that the body of the 
HTML file is beginning. Then, an arbitrary number of HTML commands 166 are 
provided to, for example, display images of the web page on the video display of the 

15 client machine. A <CENTER> command 168 will cause a centering of following objects 
with respect to the browser window on the video display of the client machine. Next, an 
<EMBED ...> command 170 of the present invention defines a force button object that 
will be displayed on the client machine. Since the <CENTER> command 168 was given 
just prior to the <EMBED ...> command, this "force button" will be centered in the 

20 displayed browser window. Other force objects besides button objects can also be 
defined and displayed, such as links, text, sliders, game objects (balls, paddles, etc.), 
avatars, windows, icons, menu bars, drop-down menus, or other objects. 

In a first line 172 of the <EMBED ...> command, the force button object is 
defined by a "IFF" extension file, namely "FORCEBUTTON.IFF." Next, in a line 174, 

25 the size of the button is indicated to be 100 pixels by 100 pixels. In a line 176, the initial 
state of the button is indicated to be "up" (i.e., unselected), and a line 178 defines the 
force effect to be "vibration." A number of parameters 180 defining the character and 
nature of the vibration are also provided (start time, length, frequency, magnitude, etc.). 
In a line 182, the "trigger" for the force effect is given by the function 

30 " MOUSE WITHIN" with its associated parameters, and by the function 
"BUTTONSTATE." The function MOUSEWITHIN determines whether the pointer, the 
position of which is controlled by the force feedback device, is within the specified 
boundaries defining a region of the force button. This region can be specified by the 
parameters and, for example, can be defined as the exact displayed area of the button, or 

35 can be defined as a sub-region within the button that is smaller than the displayed size of 
the button. The function BUTTONSTATE determines whether a button or switch of the 
force feedback device is in the desired state to trigger the force object event (e.g., a button 
event in this example). In a line 184, the icon representing the force button is specified as 
16 
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"LOUIS.GIF," and the text associated with the button is defined as "Hi, I'm Louis" in a 
line 186. The font of the text is given as "Helvetica" in a line 188. Other force effects, 
triggers and parameters can also be associated with the force object. For example, a force 
(such as a vibration) can be triggered if the pointing icon is moved a predetermined 
5 velocity or within a predefined range of velocities within the force object. Or, a trajectory 
of the pointing icon on a force object can trigger a force, like a circle gesture. 

The <EMBED ...> command is an existing functionality of HTML. It essentially 
embeds function calls which are handled by the web browser. If the suffix of the 
specified file is a known, standard suffix type, the call is executed directly by the web 

10 browser. If, however, the suffix (.IFF in this instance) is not a standard feature of the web 
browser, the browser will first look for a "plug-in" to implement this feature and, if a 
suitable plug-in is not found, it will look for application programs implementing this 
feature. In one embodiment, a plug-in including a reference to a Dynamically Linked 
Library (DLL) is provided to give functionality to the .IFF suffix. The DLL can be 

1 5 provided local to the client machine or on another linked resource. 

With continuing reference to Fig. 7b, the centering command is terminated at line 
190 with the </CENTER> command. Additional HTML commands 192 can then be 
provided, and the body of the HTML file is terminated by the </BODY> command 194. 
The end of the HTML file is indicated at 196 with the </HTML> command, i.e. this 
20 command 1 96 is the " eof ' command of the HTML file 32. 

The present invention also provides for programmability of the embedded force 
feedback object. An example of this programmability is shown at 198. This optional 
programmable command can be inserted into the EMBED command 170 and can include, 
for example, an iterative loop. In line 200, a "FOR" command initializes a counter i to 0, 

25 indicates that the counter I is incremented by one per each pass through the loop, and it 
indicates that the loop should be completed five times, i.e. while i<5. The body of the 
loop includes a command line 202 which indicates that a force feedback "vibrate" with 
associated parameters should be evoked, and a line 204 indicates that a 5 second wait 
should be provided after the vibration has occurred. This step will repeat five times, i.e. 

30 the command 198 will cause five vibration sequences separated by four 5 second pauses, 
and followed by a final 5 second pause. By providing programmability to the force 
feedback object, force feedback effects based upon past events and upon a complex 
interaction of factors can be provided. 

In Fig. 8, the "Parse And Interpret HTML Component" or simply "Process 
35 HTML Component" step 160 of Fig. 7a is illustrated in greater detail. In Fig. 8, process 
160 begins at 206 and, in a step 208, it is determined whether there is an embedded "tag" 
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for a force object, e.g. a tag having an .IFF reference. An example of the embedded tag is 
shown at the EMBED command 170 of Fig. 7b. If there is such a tag, step 210 uses the 
plug-in software of the present invention to interpret the .IFF file, and the process is 
completed at 212. Otherwise, another type of HTML command has been encountered, 
5 and the standard web browser parser and interpreter processes this HTML component in a 
step 214, after which the process is completed at 212. 

In Fig. 9, the step 210 "Plug-In Interprets .IFF File" of Fig. 8 is described in 
greater detail. Process 210 begins at 216, and in a step 218, a "framework" is created for 
the force object. The framework provides a particular set of generic features to 

10 implement the specified force object, and preferably includes no specific parameters or 
functions for the force object. Next, in a step 220, the name/value pairs are parsed and, in 
a step 222, the force object is built upon this framework based upon the name/value pairs. 
A name/value pair includes the name of a component and its associated parameters. For 
example, one name might be "BUTTONSTATE" and its value (or parameter) might be 

1 5 " UP" (or " UNSELECTED" ). The process 2 1 0 is completed at 224. 

In Fig. 9a, an image 226 to be displayed on a screen of a video monitor or other 
visual display is illustrated. More specifically, image 226 can be generated by the 
popular Netscape Navigator® web browser. The image 226 includes a window 228 
including a header portion 230 and a body portion 232. The header portion 230 includes 

20 a number of navigation buttons 234 and special purpose buttons 236 for purposes well- 
known to those familiar with the Netscape Navigator web browser. In addition, the 
header portion 230 displays the URL of the currently displayed web page at 238. In this 
instance, the URL is "http://www.immerse.com/demo." The images displayed within 
the body portion 232 of the window 228 are created by the aforementioned processing of 

25 the HTML file by the web browser. 

The area within the body portion 232 has been provided with a number of regions 
and buttons to illustrate some of the concepts of the present invention. The force 
feedback device controls the position of a pointer icon 240 which can be caused to 
interact with the various regions and buttons. As an example, when the force feedback 

30 device is manipulated by the user to cause the pointer icon 240 to move within a 
"texture" region 242, force feedback commands can be created for the force feedback 
device to provide a desired "texture" to the force feedback device. For example, the 
texture can feel "rough" to the user by causing the force feedback device to place forces 
on the user manipulatable object that emulate a rough or bumpy surface. In a region 244, 

35 a "viscosity" force feedback can be provided. With this form of force feedback, as the 
pointer icon is moved through field 244, a viscous "drag" force is emulated on the user 
manipulatable object. In a region 246, inertial forces can be felt. Therefore, a pointer 
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icon being moved through an " inertia" region would require relatively little or no force 
to move in a straight line, but would require greater forces to accelerate in a new direction 
or to be stopped. The inertial force sensations can be applied to the user manipulatable 
object and felt by the user. In a "keep out" region 248, the pointer image is prevented 
5 from entering the region. This is accomplished by creating a repulsive force on the user 
manipulatable object using a force feedback command to the force feedback device which 
prevents or inhibits the user from moving the user manipulatable object in a direction of 
the region 248 when the pointer icon 240 contacts the periphery of the region 248. In 
contrast, a " snap-in" region 250 will pull a pointer icon 240 to a center 252 whenever the 

10 pointer icon engages the periphery of the snap-in region 250 and apply a corresponding 
attractive force on the user manipulatable object. A "spring" region 243 emulates a 
spring function such that a pointer icon moving into the spring region "compresses" a 
spring, which exerts a spring force on the user manipulatable object which opposes the 
movement of the pointer icon. A region 256 is a "Force To Left" region where the 

15 pointer icon within the region 256 is forced to the left side of the region and the user 
manipulatable object is forced in a corresponding direction as if influenced by some 
invisible magnetic force or gravitational force. A region 258 illustrates that regions can 
be of any size or shape and that within a region different force effects can be developed. 
In this example, within region 258 there is a texture core 260 surrounded by a vibration 

20 ring 262. Therefore, as the pointer icon 240 moves into the region 258, the user first 
experiences vibration from the ring 262, and then experiences a texture as the pointer icon 
moves within the core 260. 

The exemplary force feedback web page of Fig. 9a is also provided with several 
force feedback buttons. In a first button 264, the placement of the pointer icon 240 over 

25 the button and the pressing of a button (i.e., a switch) on the force feedback device to 
create a "button click", "button down", or simply a "button event" input, will then 
cause a "buzz" command to be sent to the force feedback device. The buzz command 
would, for example, cause a vibration force on the user manipulatable object. Similarly, 
the selection of the "jolt" button 266 will cause a jolting force (e.g., a short-duration 

30 pulse of force) to be provided at the force feedback device, and the pressing of the 
"detent" button 268 will cause a "detent" to be created for the force feedback device. 
By "detent" it is meant that the user manipulatable object will be controlled by the force 
feedback actuators such that it feels as if a mechanical-type detent exists at the position 
that the user manipulatable object was in when the detent button 268 was activated. 

35 These and other forces resulting from a pointing icon interacting with various 

objects displayed on a computer screen are also described in co-pending patent 
application serial no. 08/571,606 filed 12/13/95, Atty. Ref. IMM1P015, the disclosure of 
which is incorporated herein by reference. 
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In Fig. 10, a process 270 of the plug-in software of the present invention is 
illustrated. The process 270 begins at 272 and, in a step 274, the position and button state 
of the force feedback device is monitored. Next, in a step 276, a force feedback 
command is created in response to the detected position and state. Finally, a command is 
5 sent to the Dynamically Linked Library (DLL) to place a force feedback command on the 
interface which can be parsed and interpreted by the force feedback device. The process 
is then completed as indicated at 280. 

It should be noted that the force feedback driver (e.g., browser plug-in or DLL) 
can have the ability to interact with JAVA code. In this embodiment, the plug-in reads 
10 and executes JAVA commands using the browser's run-time JAVA interpreter. JAVA 
can optionally be used to make "applets" which perform dynamic models, such as 
creating complex force feedback sensations. 

It should also be noted that the force feedback device itself can have a JAVA 
interpreting chip on board, permitting the plug-in driver to download JAVA code to the 
15 force feedback device to be executed on the device. JAVA and JAVA interpreting chips 
are available under license from SUN Microcomputers of Mountain View, California. 

Furthermore, the force feedback driver (e.g., browser plug-in or DLL) can have 
the ability to interact with instructions provided in other languages besides HTML. For 
example, virtual reality 3-D graphical environments are increasingly being created and 

20 implemented over the World Wide Web and Internet using languages such as the Virtual 
Reality Modeling Language (VRML) and software such as Active X available from 
Microsoft Corporation. In these 3-D graphical environments, users may interact with 
programmed 3-D objects and constructs using client computer 14 or 16, and may also 
interact with 3-D graphical representations (or "avatars") controlled by other users over 

25 the World Wide Web/Internet from other client computers. Force feedback commands 
and parameters can be provided in the instructions or files of these other protocols and 
languages and received by a client computer system in an equivalent manner to that 
described above so that force feedback can be experienced in simulated 3-D space. For 
example, embedded force feedback routines can be included in the VRML data for a 

30 virtual environment so that when the user moves into a virtual wall, an obstruction force 
is generated on the user-manipulatable object. Or, when the user carries a virtual object 
in a controlled virtual glove, the user might feel a simulated weight of the virtual object 
on the user manipulatable object. In such an embodiment, the force feedback device 
preferably provides the user with three or more degrees of freedom of movement so that 

35 input in three dimensions can be provided to the client computer. 



IMM1P053 



20 



Patent 



FIGURE 1 1 is a schematic diagram of a multi-computer network system 300 
which can be used in accordance with peer-to-peer (client-to-client) embodiments 
disclosed herein. Force feedback implementation over networks can be based on 
displayed interactions and client-to-client direct communication of force feedback 
5 information. Two or more host computer applications or graphical environments can be 
linked over a computer network to provide multi-user interactions involving two, three or 
more users. The same application programs can be running on each of the linked host 
computers, or different application programs can be linked. For example, different types 
of web browsers, each able to parse and communicate in TCP/IP protocols, can 

10 communicate with each other and display graphical objects based on position information 
received from the other computers. Forces can be output based on both information from 
a local force feedback device and application program, as well as information received 
from other host computers over the network. A bi-directional networked interface allows 
users at different connected host computers to interact in visual, auditory, and haptic 

15 ways. 

In one embodiment, a first site 310 includes computer 312 that implements a 
graphical environment, such as a web browser, simulation, or game application, and a 
first user utilizes display device 314 and force feedback interface device 316. Optionally, 
local microprocessor 318 is coupled to interface device 316 as described with reference to 

20 Figure 3. At a second site 320, computer 322 implements the graphical environment, 
display device 324 displays images to a second user, force feedback interface device 326 
interacts with the second user, and local microprocessor 328 can optionally be included. 
The first site is a "remote" site with reference to the second site, and vice versa. Each 
computer 312 and 322 implements a local application program so that each display 

25 device 314 and 324 displays a local visual environment, such as a web page, a video 
game, or a simulation. Additional users and computers that implement the graphical 
environment can be included in the network system 300 similarly to the systems 
described. In some embodiments, a graphical environment need not be displayed or is 
not updated, and forces are output to a user based on motion of the user manipulatable 

30 objects of the force feedback devices of the connected computers. The computers 312 
and 322 are connected by a computer network, which can be the Internet or other form of 
network that allows bi-directional transmission of information. 

Each local computer 312 and 322 has direct access to its own interface device 316 
and 326, respectively, but does not have direct access to the remote interface device used 
35 by the other user. Thus, the information which describes the position, orientation, other 
motion or state characteristics, button data, and other information related to each local 
interface device (collectively considered "motion/state information" herein) is conveyed 
to the other remote computer. Each local computer 312 and 322 therefore has direct 
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access to the local interface device and networked access to the motion/state information 
of the remote interface device, allowing a consistent interaction for both users. 

The computers 312 and 322 need only exchange the information that is necessary 
to update the simulated graphical objects controlled by the remote users and other 
5 simulated characteristics that may have been affected by the input of a user. This 
minimal information exchange is often necessary when using networks having low or 
limited bandwidth and which have a slow rate of information transfer, such as many 
current connections to the Internet/World Wide Web, often implemented (for many home 
computer users) using low bandwidth telephone connections and relatively low- 

10 bandwidth modems or similar telecommunication devices. The computationally- 
intensive force feedback calculations to implement the interactions between a user- 
controlled object (e.g. cursor or paddle) and other objects (e.g., icons, GUI elements, 
other paddle) are preferably handled locally. The resulting outcome of the force feedback 
calculations/interactions are transmitted to remote users so as to minimize the information 

1 5 that is transmitted to other computer systems. 

One type of information which is sent between the networked computers is 
motion/location/state information. For example, in a multi-user game interaction, when a 
local user controls one paddle and a remote user controls a different paddle, the position 
of the remote user's manipulandum is needed to determine paddle interaction and 

20 appropriate forces. Or, if a moving graphical object interacts with a paddle controlled by 
a local user, the local computer processes the interaction, generates the required local 
force feedback sensations, computes the new location and velocity of the moving 
graphical object as a result of the interaction, and conveys the new graphical information 
to the remote computer(s) so that all game applications can be re-coordinated after the 

25 object interaction. The remote computer then computes any force feedback sensations 
occurring at its own site resulting from the new object position, motion, etc. 

When using a network having low- or limited- bandwidth, there may still be a 
substantial time delay from when a local graphical object, such as a cursor or paddle, 
changes its location/motion/state information and when the remote web browsers or 

30 application programs receive and are updated with that information. Thus, a user at a 
given site may be viewing a remote-user-controlled graphical object at a time delay while 
viewing his own cursor in real time without a time delay. For example, the user may 
witness a cursor-icon interaction a few seconds after the actual event happened on the 
remote user's local implementation of the interaction. Obviously, this can cause 

35 problems in the experience of networked interactions and game play. To compensate for 
this problem, a networked graphical environment may introduce a short time delay before 
events occur locally. For example, a short delay can be implemented on the local 
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computer before a ball bounces off of a paddle to reduce the timing discontinuity between 
remote and local users. 

In addition, force feedback or "feel sensation information" can be transferred 
from one host computer to another over the network. This type of information can be 
5 provided, for example, if a force should be output that is not based on position or motion 
of the user manipulatable objects or interacting graphical objects. Thus, if a button press 
on a joystick manipulandum of force feedback device 316 designates that a vibration is to 
be output on the other joystick manipulandum of force feedback device 326, a force 
feedback command or other similar information can be sent from computer 312 to 

10 computer 322, preferably including parameters describing the vibration feel sensation. 
Computer 322 parses and interprets the command and then commands the force feedback 
device 326 to output the vibration on the joystick of device 326. Such commands and 
parameters can be implemented similarly to the HTML or VRML embodiments described 
above, or in a different format. The computer 322 thus receives the feel sensation 

15 information directly from the other computer 312. Alternatively, the computer 312 can 
simply send the button press information, so that the computer 322 interprets the button 
press as a particular force sensation and outputs that sensation. However, with such an 
embodiment, the computer 322 would need mapping information that indicates which 
feel sensation corresponds to the received button press, and this mapping information has 

20 to be updated periodically so as to provide synchronization. If force feedback 
information is sent directly, there is no need for the computer 322 and/or force feedback 
device 326 to store data mapping a particular button press to a feel sensation, saving 
memory and synchronization steps. Such feel sensation information can also be useful to 
characterize graphical objects in a game or simulation which one computer generates or 

25 updates and needs to convey to any other linked computers to provide synchronization. 
For example, a wall graphical object can be characterized as having a hard or soft surface, 
having a smooth or frictional surface, and having other force characteristics. 

The force feedback command or other information relayed from a host computer 
to its force feedback device can be determined based on the motion/state information 

30 and/or force feedback information received over the network. In many cases, the force 
feedback command or other information can be also based on input from the local force 
feedback device. For example, a force need not be commanded until a controlled 
graphical object impacts a different graphical object. To determine whether the user- 
controlled graphical object has impacted another object, position (or other motion) 

35 information is received from the local force feedback device which indicates the current 
position of the user object in its degrees of freedom. From this information, a new 
position of the user-controlled graphical object is determined, and any interactions of this 
object with other objects are realized. 
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Many different applications of force feedback implementation over networks in a 
client-to-client configuration can be implemented. For example, two graphical objects, 
each controlled by a different user, can interact and the users can experience forces based 
on the interaction. In one implementation, a first computer displays a first user controlled 
5 graphical object that is moved in conjunction with the first user's manipulation of a first 
force feedback device connected to the first computer. The first computer also displays a 
second graphical object. The second computer, connected to the first computer by a 
network, also displays the first and second graphical objects on a display screen of a 
second host computer. The second graphical object is moved in conjunction with the 
10 second user's manipulation of a second force feedback device connected to the second 
computer. Force feedback can be provided to both first and second computers based on 
the interaction of the first and second object. 

One example of using client-to-client interaction in a game is shown in FIGURE 
12a. A 2-D implementation of displayed graphical objects on display device 64 is shown. 

15 Paddle 360 can be controlled by a first host computer system, such as client machine 14, 
and paddle 362 can be controlled by a second host computer system, such as client 
machine 16. Ball 352 can be moved on display screen 64 according to simulated physical 
parameters, such as velocity, acceleration, gravity, compliance of objects, and other 
parameters as discussed previously. When the ball 352 collides with paddle 362, the 

20 paddle flexes, and the user feels the collision force. For example, if ball 352 is moving in 
direction 364, then the user feels a force in the equivalent degrees of freedom of user 
object 76. In some embodiments, both the paddle 362 and the ball 364 can be moved in 
direction 364 to simulate the paddle being pushed back by the ball. FIGURE 12b shows 
a similar embodiment in which a perspective view (or simulated 3-D view) of the 

25 graphical objects is shown on display screen 20. 

The user can also move the user object so that the paddle moves in a direction 
366. The user will thus feel like he or she is "carrying" the weight of the ball, as in a 
sling. The ball will then be released from the paddle and move toward the other paddle 
360. As is well known, a goal in such a game might be to direct the ball into the 

30 opposing goal. Thus, the first user can try to direct the ball into goal 368, and the second 
user can control paddle 360 to direct the ball into goal 370. Paddles 360 and 362 are used 
to block the ball from moving into the defended goal and to direct the ball back at the 
desired goal. By moving the paddle in a combination of direction 366 and up and down 
movement, the user can influence the movement of the ball to a fine degree, thus 

35 allowing a player's skill to influence game results to a greater degree than in previous 
games without force feedback. In addition, other features can be included to further 
influence the ball's direction and the forces felt by the user. For example, the orientation 
of the paddle can be changed by rotating the paddle about a center point of the paddle, 
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and force feedback can be appropriately applied in that degree of freedom. Other features 
can also be provided, such as allowing a ball to "stick" to a paddle when the two objects 
collide and/or when a button is pressed by the user. The user could then activate the 
button, for example, to release the ball at a desired time. 

5 Each player can feel the forces on their respective paddle from the ball directed by 

the other player. In addition, if the two paddles 360 and 362 were brought into contact 
with one another, each player can feel the direct force of the other player on each player's 
user object. That is, the first user's force on his user object causes his paddle 362 to move 
into the other paddle 360, which would cause both the first and second users to feel the 
10 collision force. If the first paddle 362 were allowed to push the other paddle 360 across 
the screen, then the second user would feel the first user's pushing force. The first user 
would feel similar forces from the second user. 

In a "tug of war" game example, the first and second graphical objects (such as 
two paddles or other objects) can be visually connected. When the first user controls the 

15 first graphical object to move left, this position information is transferred to the second 
computer, and the second user controlling the second graphical object feels a force in a 
left direction resulting from the first player's manipulation. A similar result occurs for 
the first player when the second player manipulates the second graphical object. This 
creates the effect as if each player were pushing the other player directly. Furthermore, 

20 force information ("feel sensation information") can also be transmitted between 
computers. For example, if a flexible "rope" is modelled connecting the first and second 
graphical objects, and the first user manipulates the first force feedback device so that the 
rope is made to oscillate or vibrate, then the first computer can send feel sensation 
information to the second computer that informs the second computer to command a 

25 vibration feel sensation on the second force feedback device, with appropriate parameters 
describing the vibration such as frequency, amplitude, and duration. The second user 
thus immediately feels appropriate forces caused by the first user. The winner of the tug- 
of-war can be the first user to move his or her graphical object to a specific goal or 
displayed location in opposition to forces from the other player and/or other obstacles. 

30 Alternatively, a user can be designated to win the tug-of-war or other game if that user 
can maintain a particular position of the user manipulatable object amid the forces output 
based on the interaction of the graphical objects and caused by both users. 

A different example of a client-to-client communication of force feedback 
information can take the form of a "massage" interface. Two users, for example, can 
35 interact with each other by each feeling the presence of the other in an "intimate" way 
through the use of forces influenced by the device manipulation of the other user. The 
"input" of the user at one client computer is felt as "output" by the user at the other 
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client computer, and vice-versa, much like the tug-of-war example above. Referring to 
Fig. 1, a first user can be interfaced to a network, such as the Internet, through client 
machine 14 using a force feedback massage interface. A second user can be interfaced to 
the network through client machine 16, also using a force feedback massage interface. 
5 For example, the interface can include a user manipulatable object that is a representative 
body part such as a hand, foot, sexual organ, etc. (moveable in one or more degrees of 
freedom). The first massage interface can apply conditions, effects, or other force 
sensations to the first user, depending upon the feel sensation information sent from the 
second client machine. For example, the first user can press a button that causes the first 

10 client to send a vibration command to the second client and second user via the network 
connection. The first user or first client can specify the magnitude, frequency, wave 
shape, and/or the direction of the vibration sensation. In addition, the force feedback 
massage interface of the first user can receive information that is generated in part by the 
force feedback massage interface of the second user. For example, client machine 14 can 

15 automatically determine that feel sensation parameters (magnitude, direction, frequency, 
etc.) are sent by interpreting the positional input that the first user generates using the first 
force feedback device. Thus, transferring physical information back and forth allows the 
first and second users to interact with each other through the network. The physical 
sensations are exchanged by performing two steps: identifying a client (e.g., by URL) and 

20 sending feel sensation information that can be interpreted by that client and 
displayed/output on the force feedback interface device. 

In a simple application of such an embodiment, the first user can massage the 
back of the second user by linking the first force feedback device connected to the first 
client machine with the second force feedback device connected to the second client 

25 machine. A user manipulatable object of the first force feedback device can be grasped or 
otherwise physically contacted by the hand of the first user, tracking the motion of the 
user's hand and outputting forces to the user's hand. The second force feedback device 
can have a user manipulatable object shaped like a hand that can engage the back of the 
second user. The motion of the user object of the first force feedback device can be 

30 linked to the motion of the hand object of the second force feedback device such that 
when the first user moves his or her hand, the hand object connected to the second client 
machine moves around and engages the back of the second user. Using this embodiment, 
the first user can massage the back of the second user, where the second user feels the 
hand object with a motion and pressure dependent on the first user's input. The first user 

35 also receives force feedback from the interaction of the hand object with the back of the 
second user, where the pressure and motion from the second user's back applied to the 
hand object is relayed to the user object held by the first user. Thus, if the second user 
leans back and applies force on the hand object, the first user feels that force. 
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In addition to the feel sensation information sent between the first client and the 
second client, other sensory information can be sent and displayed. For example, audio 
and video information can be transferred between the client computers. For example, 
both client computers can be connected to video cameras and microphones. The data 
5 from the video and microphone used by the first user can be sent to and displayed to the 
second user via a display device, such as a video screen, and via an audio output device, 
such as speakers; and the data from the second user can likewise be displayed to the first 
user. These other modalities can complete an environment that allows two or more users 
to interact through sight, sound, and touch. 

10 In one embodiment of a client-to-client interaction, the force feedback device used 

by the first user can be linked to the force feedback device used by the second user such 
that the motion of the two user manipulatable objects are desired to be synchronized. If 
either user pushes his or her device such that it diverges from synchronization, a restoring 
force such as a spring force, for example, can be applied in the direction of the diverging 

15 device (or the non-diverging device, or both devices, as desired) needed to restore 
synchronization. The restoring force is applied as long as the two user manipulatable 
objects are out of synchronization. To determine if the user objects are in 
synchronization, the positions of the two user objects can be compared, where the 
position of each remote manipulandum is sent across the network to the other client to 

20 allow the comparison. For example, if the two positions maintain a constant relative 
distance between them within a predefined threshold distance, then no restoring force 
need be applied. If graphical objects are displayed based on the two user object positions, 
the graphical objects in some embodiments can be maintained in visual synchronization 
even though the user manipulatable objects are no longer in actual positional 

25 synchronization (if such visual synchronization is desired). 

Many types of feel sensations can be sent across the network and combined in 
various ways. For example, a constant force or a spring force can be commanded to be 
applied to the force feedback interface device over the network, and other feel 
sensations/forces such as vibrations sensations can also be commanded over the network 

30 to be simultaneously overlaid on the constant or spring force. For example, the first user 
can press a button that causes the force feedback massage interface of the second user to 
output a vibration sensation over any forces already being experienced. A user can 
design a feel sensation using a feel sensation editor, such as shown in co-pending patent 
applications serial nos. 08/846,011 and 08/877,114, both assigned to the assignee of the 

35 present application and incorporated herein by reference. This allows the users to design, 
for example, a massage vibration sensation - including magnitude, direction, envelope, 
waveform, and to send the created sensation to a different user's site to be experienced by 
the user or used in that user's own force sensations. In addition, it should be noted that a 
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single client can be interfaced to multiple clients such that a force sensation sent from one 
client is received by many clients over the network. 

The first and second client computers can also be directly connected though a 
phone line or other transmission medium. The client computers can be networked 
5 through a direct TCP/IP connection or other LAN connection, or can be connected 
through the Internet as described above. For example, both client computers can be 
connected to the same server that provides a web page or other information to the clients. 
Information can be provided from one client to the server, then from the server to the 
desired client computer (or, a server can perform processing on data before it is sent to 

10 the receiving client computer). Alternatively, the clients can be connected directly over 
the Internet with data provided directly between clients (or as directly as possible over the 
distributed Internet), which tends to reduce the time delay of transmission. The clients 
need not send visual page information or feel sensation information to each other unless 
the interaction of one user causes the visual display to change on the other user's visual 

15 display. 

In yet another embodiment, the peer-to-peer embodiments described above can 
also include communications with a server machine 18, such as shown in Fig. 1. For 
example, before sending feel sensation information from a first client to a second client, 
the first client can access at least part of the feel sensation information from the server 

20 machine 18. If, for example, server machine 18 provides a web page for access, the first 
client can download the web page when the server is accessed and the user can select a 
particular feel sensation displayed on the web page to be downloaded to download the 
feel sensation to the first client, or a feel sensation can be automatically downloaded upon 
establishment of a client-server connection. Alternatively, feel sensation information 

25 such as the commands and parameters disclosed above can be sent to the first client 
embedded in the web page. When the user of the first client commands that the feel 
sensation be output (such as by pressing a button, etc.), then the first client can transfer 
the feel sensation information to the second client to be output by the force feedback 
device of the second client. Alternatively, the server computer can send the feel sensation 

30 information (e.g., which may have been selected by the first client) directly to the second 
client if such a connection has been established. In one embodiment, the feel sensation 
information can be embedded in an HTML file as described in above embodiments, or 
the feel sensation information can be provided on its own in a similar format to the 
embedded reference disclosed above (i.e. a force object can be created locally from 

35 commands, parameters, or other references received by a client). The feel sensation 
information can also be provided in other formats. 
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While this invention has been described in terms of several preferred 
embodiments, there are alterations, permutations, and equivalents which fall within the 
scope of this invention. It should also be noted that there are may alternative ways of 
implementing both the process and apparatus of the present invention. It is therefore 
intended that the following appended claims be interpreted as including all such 
alterations, permutations, and equivalents as fall within the true spirit and scope of the 
present invention. 

What is claimed is: 
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CLAIMS 



1. A networked force feedback system comprising: 

5 a network; 

a first computer coupled to said network, said first computer including a first 
visual display and a first interface device capable of providing a first computer input, said 
first interface device including at least one actuator capable of providing force feedback 
in response to a force feedback signal provided by said first computer, said first computer 
10 developing an image on said visual display that is associated with stored force feedback 
information, wherein said first computer produces said image and said force feedback 
signal based on information received from a second computer; and 

a second computer coupled to said network, said second computer including a 
second visual display and a second interface device capable of providing a second 

15 computer input, said second interface device including at least one actuator capable of 
providing force feedback in response to a force feedback signal provided by said second 
computer, said second computer developing an image on said visual display that is 
associated with stored force feedback information, wherein said second computer 
produces said image and said force feedback signal based on information received from 

20 said first computer. 

2. A networked force feedback system as recited in claim 1 wherein said first 
computer produces said image and said force feedback signal based on information 
received from a second computer and based on said first computer input, and wherein 
said second computer produces said image and said force feedback signal based on 

25 information received from said first computer and based on said second computer input. 

3. A networked force feedback system as recited in claim 1 wherein both said 
first computer and said second computer are network access computers which 
communicate over said network using TCP/IP protocols. 

4. A networked force feedback system as recited in claim 3 wherein said first 
30 computer sends information to a Uniform Resource Locator of said second computer. 

5. A networked force feedback system as recited in claim 1 wherein said 
human/computer interface device coupled to said second computer includes a local 
microprocessor that communicates with said second computer, a plurality of actuators for 
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providing a force feedback, and at least one sensor for sensing positions of said 
human/computer interface device. 

6. A networked force feedback system as recited in claim 5 wherein said 
human/computer interface device coupled to said second computer includes a user 

5 manipulatable object for receiving input from said user, said user manipulatable object 
being movable in two linear degrees of freedom. 

7. A networked force feedback system as recited in claim 6 wherein said user 
manipulatable object is receptive to a finger of said user for manipulating said user 
manipulatable object in said two linear degrees of freedom. 

10 8. A networked force feedback system as recited in claim 5 wherein said 

force feedback signal includes a force feedback command that can be parsed by said local 
microprocessor such that said microprocessor can control said actuators in response to 
said force feedback command in a control loop with said sensors. 

15 9. A networked force feedback system comprising: 

network means; 

first computer means coupled to said network means; and 

second computer means coupled to said network means, said second computer 
means including visual display and human/computer interface means, said second 

20 computer means further including means for displaying an image on said visual display 
means, said second computer means further including means for interpreting said visual 
information and feel sensation information transferred to said second computer means 
from said first computer means over said network means, displaying said image from said 
visual information, and associating said feel sensation information with said visual 

25 information, said second computer means further including means for developing 
physical feel sensations at said human/computer interface means using said feel sensation 
information based on said visual information and a second computer input provided by 
said human/computer interface means, wherein said physical feel sensations are produced 
using actuator means included in said human/computer interface means. 

30 

10. A networked force feedback system as recited in claim 9 wherein said 
means for developing physical force feedback uses said feel sensation information in 
response to a graphical object controlled by said human/computer interface means 
interacting with a different graphical object of said image. 
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11. A networked force feedback system as recited in claim 9 wherein both said 
first computer means and said second computer means are network access computer 
means which communicate over said network using TCP/IP protocols. 

12. A networked force feedback system as recited in claim 9 wherein said 
5 means for displaying an image includes means for displaying a graphical environment, 

wherein said graphical environment includes said visual information transferred from said 
first computer means to said second computer means. 

13. A networked force feedback system as recited in claim 12 wherein said 
second computer means input comprises at least one of a position input for said 

1 0 human/computer interface device, and a button click input. 

14. A networked force feedback system as recited in claim 9 wherein said 
human/computer interface means coupled to said second computer means includes a local 
microprocessor means that communicates with said second computer means, a plurality 
of said actuator means for providing said physical feel sensations, and a plurality of 

1 5 sensor means for sensing positions of said human/computer interface means. 

15. A networked force feedback system as recited in claim 14 wherein said 
second computer means sends a force feedback command to said local microprocessor 
means that can be parsed by said local microprocessor means such that said 
microprocessor means can control said actuator means in response to said force feedback 

20 command in a control loop with said sensor means. 

16. A method for providing force feedback between a first computer and a second 
computer, the first computer including a first force feedback device providing computer- 
controlled physical force feedback to a first user of said first force feedback device, the 

25 second computer including a visual display and a second force feedback device providing 
computer-controlled physical force feedback to a second user of said second force 
feedback device, the method comprising: 

establishing a connection between said first computer and said second computer 
over a network; 

30 developing an image on said visual display of said second computer; 

receiving first computer information at said second computer from said first 
computer over said network; 
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receiving input information at said second computer from said second force 
feedback device in response to manipulation of said second force feedback device by said 
user; and 

providing a force feedback signal to said second force feedback device from said 
5 second computer, said force feedback signal being based on said first computer 
information from said first computer and said input information from said second force 
feedback device, wherein said force feedback signal causes said second force feedback 
device to output force feedback using an actuator of said second force feedback device. 



10 17. A method as recited in claim 16 wherein said first computer information 

includes input information from said first force feedback device representing a position of 
a user manipulatable object of said first force feedback device. 

18. A method as recited in claim 17 wherein said first computer information 
includes force feedback information indicating a force sensation to be output by said 

15 second force feedback device. 

19. A method as recited in claim 16 further comprising sending second computer 
information from said second computer to said first computer over said network. 

20. A method as recited in claim 19 wherein said second computer information 
includes said input information from said second force feedback device and force 

20 feedback information indicating a force sensation to be output by said first force feedback 
device. 

21. A method as recited in claim 16 wherein said image includes displaying a 
first graphical object controlled by a user of said first force feedback device, and 
displaying a second graphical object controlled by a user of said second force feedback 

25 device. 

22. A method as recited in claim 21 wherein said first and second graphical 
objects are paddles. 

23. A method as recited in claim 21 wherein said first and second graphical 
objects are displayed in a web page. 

30 24. A method as recited in claim 16 wherein said first force feedback device 

includes an object representing a body part to be physically contacted by a user. 
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25. A method as recited in claim 16 wherein said second force feedback device 
includes a local microprocessor that communicates with said second computer, wherein 
said local microprocessor parses a force feedback command sent by said second computer 
such that said local microprocessor can control said actuator in response to said force 
5 feedback command in a control loop with at least one sensor of said second force 
feedback device. 



26. A method for allowing two users to interact physically over a computer 
network, wherein a first manipulandum is physically contacted and moved by a first user 

10 in at least one degree of freedom and a second manipulandum is physically contacted and 
moved by a second user in at least one degree of freedom, the method comprising: 

transmitting first information including an indication of said movement of said 
first manipulandum over said computer network to said second manipulandum physically 
contacted by said second user; 

15 applying a force to said second manipulandum based on said indication of 

movement of said first manipulandum such that said second user feels an interaction 
based on movement of said first manipulandum; 

transmitting second information including an indication of movement of said 
second manipulandum over said computer network to said first manipulandum; and 

20 applying a force to said first manipulandum based on said indication of movement 

of said second manipulandum such that said first user feels an interaction based on 
movement of said second manipulandum. 

27. A method as recited in claim 26 wherein said first and second 
manipulandums are coupled to first and second computers, respectively, that are coupled 

25 to said computer network. 

28. A method as recited in claim 27 further comprising developing an image on a 
visual display of said first and second computers, said image portraying a graphical 
environment at least partially responsive to said movement of said first manipulandum or 
said second manipulandum. 
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29. A method as recited in claim 28 wherein said graphical environment includes 
a first graphical object controlled by said first manipulandum and a second graphical 
object controlled by said second manipulandum, and wherein when said first and second 
graphical objects interact in said graphical object, forces are applied to said first 

5 manipulandum and said second manipulandum. 

30. A method as recited in claim 27 wherein at least one of said first and second 
manipulandums represents a body part. 

31. A method as recited in claim 27 wherein each of said first and second 
information includes feel sensation information indicating a type of force sensation to be 

10 output. 

32. A method as recited in claim 31 wherein said feel sensation information 
includes a command to overlay a designated feel sensation on a different feel sensation 
that is simultaneously output. 

33. A method as recited in claim 27 wherein said first manipulandum and said 
15 second manipulandum are each included in a force feedback device, said force feedback 

device including a local microprocessor parsing commands from one of said first and 
second computers. 

34. A method as recited in claim 33 wherein said force feedback devices each 
include at least one sensor for determining a position of said manipulandum of said force 

20 feedback device, and at least one actuator for outputting a force in a degree of freedom of 
said manipulandum of said force feedback device. 



35. A method for providing force feedback between a first computer and a second 
computer, the first computer including a first force feedback device providing computer- 
25 controlled physical force feedback to a first user of said first force feedback device, the 
second computer including a visual display and a second force feedback device providing 
computer-controlled physical force feedback to a second user of said second force 
feedback device, the method comprising: 

establishing a connection between said first computer and said second computer 
30 over a network; 
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sending first computer information to said second computer from said first 
computer over said network; 

providing a force feedback signal to said second force feedback device from said 
second computer, said force feedback signal being based on said first computer 
5 information, wherein said force feedback signal causes said second force feedback device 
to output forces to said second user using an actuator of said second force feedback 
device; 

sending second computer information to said first computer from said second 
computer over said network; 

10 providing a force feedback signal to said first force feedback device from said first 

computer, said force feedback signal being based on said second computer information, 
wherein said force feedback signal causes said first force feedback device to output forces 
to said first user using an actuator of said first force feedback device. 



15 36. A method as recited in claim 35 wherein said first computer receives input 

information from said first force feedback device in response to manipulation of said first 
force feedback device by said first user, and wherein said second computer receives input 
information from said second force feedback device in response to manipulation of said 
second force feedback device by said second user. 

20 37. A method as recited in claim 36 wherein said force feedback signal from said 

first and second computers is based on said input information from said first and second 
force feedback devices, respectively. 

38. A method as recited in claim 35 wherein said first computer information 
includes position information describing a position of a manipulandum of said first force 

25 feedback device, and wherein said second computer information includes position 
information describing a position of a manipulandum of said second force feedback 
device. 

39. A method as recited in claim 38 wherein said first computer information 
includes force feedback information indicating a force sensation to be output by said 

30 second force feedback device, and wherein said second computer information includes 
force feedback information indicating a force sensation to be output by said second force 
feedback device. 



IMM1P053 



36 



Patent 



40. A method as recited in claim 39 wherein said first and second computers each 
display a graphical environment having a first graphical object controlled by said first 
user and a second graphical object controlled by said second user. 

41. A method as recited in claim 35 wherein said first and second computers 
communicate using a TCP/IP protocol. 

42. A method as recited in claim 35 further comprising accessing a server 
computer with one of said first and second computers and downloading feel sensation 
information from said server computer, said feel sensation information to be included in 
said first computer information or said second computer information. 

43. A method as recited in claim 42 wherein said server computer provides a web 
page downloaded to said computer accessing said server, said web page including 
embedded feel sensation information. 
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Abstract of the Disclosure 



Method and apparatus for providing peer-to-peer force feedback over a computer 
network. A network force feedback system includes a network, a first computer coupled 
5 to the network, and a second computer coupled to the network. The first and second 
computers each include a visual display and a force feedback interface device. Each 
computer provides a force feedback signal to its force feedback device based on 
information received from the other, remote computer and in some cases also based on 
input from the local force feedback device. Positional information of each force feedback 
10 device and/or feel sensation information can be transmitted between the computers over 
the network. A graphical environment can be displayed to enhance the interaction 
between users. The present invention therefore permits two computer users to interact 
using force feedback provided over a network on a peer-to-peer basis. 
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